Watermark embedding and retrieval

ABSTRACT

Fingerprinting is a technique to identify multimedia signals by extracting robust perceptual features of the signal contents and searching the extracted features in a database where titles, artists, etc. are stored. Watermarking is a technique of embedding payload data in a signal in an unobtrusive manner. The invention combines both techniques. A fingerprint (M(i)) is extracted ( 1 ) from a host signal (x) and stored in a database ( 5 ). A watermark (w(i)) is embedded ( 2 ) in the host signal by modifying ( 21 ) the signal, such that the fingerprint (M′(i)) of the modified signal (x′) differs slightly from the original fingerprint. The difference is so small that the database considers them to be similar. At the receiver end, the fingerprint is extracted ( 3 ) from the watermarked signal and applied to the database, which responds by returning the original fingerprint of the signal. The embedded watermark is retrieved by subtracting ( 4 ) the original fingerprint supplied by the database from the fingerprint extracted from the host signal.

FIELD OF THE INVENTION

The invention relates to a method and arrangement for embedding awatermark in an information host signal. The invention also relates to amethod and arrangement for retrieving the embedded watermark.

BACKGROUND OF THE INVENTION

A prior-art method of embedding and retrieving a watermark is disclosedin Khalid A. Kaabneh and Abdou Youssef “Muteness-Based AudioWatermarking Technique”, presented at the 2001 IEEE Conference ondistributed computing systems workshop, pp. 379-383. In this prior-artdocument, a sequence of periods of silence (mute periods) that occur inan audio signal are extracted from the host audio signal. The watermarkis embedded by slightly modifying said mute periods, such that thewatermarked signal is not perceptually different for the human auditorysystem. The watermark is retrieved by extracting the mute periods from awatermarked copy, and subtracting the original mute periods therefrom.The difference between both is the watermark to be retrieved.

The prior-art document discloses that it is sufficient for the watermarkdetector to have the original lengths of the mute periods available. Theoriginal signal itself is not required. The document does not disclosehow the original lengths of mute periods of a watermarked signal areobtained. This is a problem in practical situations where the hostsignal is often unknown.

OBJECT AND SUMMARY OF THE INVENTION

It is an object of the invention to provide a solution for theabove-mentioned problem of the prior-art watermark embedding andretrieving method.

To this end, the invention provides a method and arrangement forembedding a watermark in a host signal, the method comprising the stepsof extracting from the host signal an original fingerprint representinga perceptual feature of the host signal, storing the originalfingerprint of the host signal in a database along with originalfingerprints of further host signals, and modifying the host signal,such that the fingerprint is modified in accordance with the watermarkto be embedded.

A fingerprint, often also referred to as signature or hash, is theresult of a function that maps perceptual features of a signal to abinary sequence. Unlike cryptographic hashes that are extremely fragile(flipping a single bit of the source data will generally result in acompletely different hash), fingerprints are herein understood to berobust. That is, if source signals are perceptually similar, then thecorresponding fingerprints are also very similar. Fingerprints aretherefore used to identify audiovisual contents. An example of suchfingerprints and use thereof is disclosed in Jaap Haitsma, Ton Kalkerand Job Oostveen: “Robust Audio Hashing for Content Identification”,published at the Content-Based Multimedia Indexing (CBMI) 2001.Conference in Brescia, Italy.

The invention is based on the recognition that the perceptual featuresbeing modified to represent the watermark as taught by the prior-artconstitutes a fingerprint. The invention exploits the insight that thefingerprint of the watermarked signal and the original fingerprint willonly slightly differ, whereas fingerprints of perceptually differenthost signals will differ substantially. It is thus achieved with theinvention that the watermarked signal can be identified by virtue of itsoriginal fingerprint stored in the database.

The corresponding method of retrieving a watermark embedded in awatermarked host signal comprises the steps of extracting from thewatermarked host signal a fingerprint representing a perceptual featureof the watermarked host signal, searching, in a database in whichoriginal fingerprints of a plurality of host signals are stored, anoriginal fingerprint which substantially matches the extractedfingerprint, and determining the difference between the originalfingerprint found in the database and said extracted fingerprint, wheresaid difference represents the watermark to be retrieved.

An embodiment of the watermark embedding method further comprises thesteps of extracting from said host signal an identification of the hostsignal, and storing said identification in the database to identify thelocation of the corresponding original fingerprint in the database. Suchan additional identification makes the process of searching the originalfingerprint in the database easier, faster and more univocal.

In an advantageous embodiment, the identification is a furtherfingerprint representing a further perceptual feature of the hostsignal. With such an embodiment, one fingerprint conveys the watermarkwhereas the other fingerprint is used to identify the location of theoriginal fingerprint in the database. Different fingerprint schemes cannow be chosen that are optimal for their respective purposes.

These and other aspects of the invention are apparent from and will beelucidated with reference to the embodiment described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a system comprising a watermarkembedding arrangement and a watermark retrieving arrangement accordingto the invention.

FIG. 2 is a schematic diagram of a further embodiment of the systemaccording to the invention.

DESCRIPTION OF EMBODIMENTS

FIG. 1 is a schematic diagram of a system comprising a watermarkembedding arrangement and a watermark retrieving arrangement accordingto the invention. The watermark embedding arrangement comprises afingerprint extracting circuit 1 and a watermark embedding module 2. Thewatermark retrieving arrangement comprises a fingerprint extractingcircuit 3 and a watermark retrieving module 4. Both arrangements haveaccess to a database 5.

The watermark embedding arrangement receives a multimedia host signal x.It will be assumed here that the multimedia signal x is an audio signal,but it will be appreciated that the multimedia signal may also be avideo, image, or other type of signal. The signal x is applied to thefingerprint extraction circuit 1, which analyses the audio signal foroccurrences of periods of silence in accordance with the teaching ofKaabneh et al. referenced herein before. The output of the extractioncircuit constitutes a fingerprint in the form of a sequence of muteperiods M(i). The fingerprint M(i) extracted from the original hostsignal x will further be referred to as original fingerprint. Theoriginal fingerprint M(i) is stored in the database 5, possibly alongwith other data such as title of the song, artist, dates of creation,etc. A plurality of fingerprints extracted from a plurality of differentsongs are stored in the database in this manner.

The audio signal x is also applied to the watermark embedding module 2.In this module, a watermark is embedded which is applied to the modulein the form of a sequence of watermark signal samples w(i). Thewatermark embedding module 2 modifies the signal x in such a manner thatthe mute periods M(i) are slightly lengthened or shortened in accordancewith the watermark. This operation can be typically expressed as:M′(i)=M(i)+w(i)  (1)

In the Figure, the watermark embedding operation is schematicallyillustrated as being carried out by a modification circuit 21, aseparation circuit 22 and an insertion circuit 23. The modificationcircuit 21 modifies the original mute periods M(i) into modified muteperiods M′(i) in accordance with respective samples w(i) of thewatermark. The watermark is embedded by first separating the originalmute periods from the signal and subsequently inserting the new muteperiods M′(i) into the signal.

The watermark retrieving arrangement receives the watermarked signal x′.The watermarked signal is applied to a fingerprint extracting circuit 3,which is similar to the fingerprint extraction circuit 1 of theembedding arrangement. Accordingly, the mute periods M′(i) are extractedfrom the watermarked signal x′. The sequence of mute periods M′(i) isapplied to the database 5 for matching with the plurality of originalfingerprints stored in the database. Since the difference between themute periods M′(i) of the watermarked signal x′ and the originalfingerprint M(i) of the original signal x is small, the most resemblingsequence of mute periods found in the database may be assumed to be theone that corresponds to the original signal x. The stored fingerprintsof other signals will generally be substantially different.

The database 5 outputs the original sequence of mute periods M′(i) tothe watermark retrieving module 4. Herein, the original mute periodsM(i) as found in the database are compared with the extracted muteperiods M′(i) to retrieve the watermark. In conformity with equation(1), this operation can be typically expressed as:w(i)=M′(i)−M(i)  (2)Additional data associated with the song, such as title of the song,artist, dates of creation, etc. can also be retrieved from the database.

In the example described above, the fingerprint is represented by themute periods of an audio signal. Another example of a robust perceptualproperty that can constitute the fingerprint is a binary representationof the energy differences of several frequency bands of the audio signalas disclosed in Haitsma et al. referenced hereinbefore. The Haitsma etal. reference also provides a detailed description of a practicalmatching strategy to find the most resembling fingerprint in thedatabase. For video and image applications, a sequence of mean luminancedifferences between image blocks may constitute the fingerprint.

FIG. 2 is a schematic diagram of a further embodiment of the system. Thesame reference numerals are used in this Figure to denote the sameelements as in FIG. 1. The embedding arrangement now comprises anidentification circuit 6 which generates an identification signal ID toidentify the original host signal x. The identification signal ID isstored in the database 5 along with the fingerprint M(i) with which itis associated. The retrieving arrangement comprises an identical orsimilar identification circuit 7, which generates an identificationsignal ID′ to identify the watermarked signal x′. In this embodiment,the identification signal ID′ is used to search in the database thestored fingerprint M(i) of original signal x.

If the original signal x is a digital computer file (e.g. an mp3 song)which includes metadata such as an ID tag identifying the file, then theidentification signal ID may take the form of (a part of) said ID tag.Assuming that the ID tag is not affected by the watermark embeddingprocess, the identification signal ID′ derived from the watermarkedsignal is exactly the same. The fairly complicated process offingerprint matching is now reduced to a simple look-up operation. Theoriginal fingerprint M(i) of the original signal can thus easily befound in the database.

If the original signal x does not have such a tag, the identificationsignal ID can be another fingerprint of the signal, now being extractedon the basis of a different robust perceptual feature of the contents.In this case, the identification circuits 6 and 7 are furtherfingerprint extraction circuits. This embodiment allows the fingerprintextraction circuits 1 and 3 for conveying the watermark, on the onehand, and identification circuits 6 and 7 for signal identification, onthe other hand, to be optimized for their respective purposes.

It should be noted that the location of the database 5 is not relevantto the invention. The database may be located at the embeddingarrangement end, the retrieving arrangement end, or remote from both.Where necessary, networks such as the Internet, may be used to applyfingerprints to or retrieve fingerprints from the database.

The invention can be summarized as follows. Fingerprinting is atechnique to identify multimedia signals by extracting robust perceptualfeatures of the signal contents and searching the extracted features ina database where titles, artists, etc. are stored. Watermarking is atechnique of embedding payload data in a signal in an unobtrusivemanner. The invention combines both techniques. A fingerprint (M(i)) isextracted (1) from a host signal (x) and stored in a database (5). Awatermark (w(i)) is embedded (2) in the host signal by modifying (21)the signal, such that the fingerprint (M′(i)) of the modified signal(x′) differs slightly from the original fingerprint. The difference isso small that the database considers them to be similar. At the receiverend, the fingerprint is extracted (3) from the watermarked signal andapplied to the database, which responds by returning the originalfingerprint of the signal. The embedded watermark is retrieved bysubtracting (4) the original fingerprint supplied by the database fromthe fingerprint extracted from the host signal.

1. A method of embedding a watermark in a host signal, comprising thesteps of: extracting from the host signal an original fingerprintrepresenting a perceptual feature of the host signal, storing theoriginal fingerprint of the host signal in a database along withoriginal fingerprints of further host signals, and modifying the hostsignal, such that the fingerprint is modified in accordance with thewatermark to be embedded.
 2. A method as claimed in claim 1, furthercomprising the steps of extracting from said host signal anidentification of the host signal, and storing said identification inthe database to identify the location of the corresponding originalfingerprint in the database.
 3. A method as claimed in claim 2, whereinsaid identification is a further fingerprint representing a furtherperceptual feature of the host signal.
 4. A method of retrieving awatermark embedded in a watermarked host signal, comprising the stepsof: extracting from the watermarked host signal a fingerprintrepresenting a perceptual feature of the watermarked host signal,searching, in a database in which original fingerprints of a pluralityof host signals are stored, an original fingerprint which substantiallymatches the extracted fingerprint, and determining the differencebetween the original fingerprint found in the database and saidextracted fingerprint, where said difference represents the watermark tobe retrieved.
 5. A method as claimed in claim 4, wherein for eachplurality of host signals an identification is stored in the database toidentify the location of the corresponding original fingerprint, themethod further comprising the steps of extracting from the watermarkedhost signal an identification of the watermarked host signal, and usingsaid identification to identify in the database the location of thecorresponding original fingerprint.
 6. A method as claimed in claim 5,wherein said identification is a further fingerprint representing afurther perceptual feature of the watermarked host signal.
 7. Anarrangement for embedding a watermark in a host signal (x), comprising:means (1) for extracting from the host signal an original fingerprint(M(i)) representing a perceptual feature of the host signal, and storingthe original fingerprint of the host signal in a database (5) along withoriginal fingerprints of further host signals, and means (2) formodifying the host signal, such that the fingerprint is modified inaccordance with the watermark (w(i)) to be embedded.
 8. An arrangementfor retrieving a watermark embedded in a watermarked host signal (x′),comprising: means (3) for extracting from the watermarked host signal afingerprint (M′(i)) representing a perceptual feature of the watermarkedhost signal, means (5) for searching in a database in which originalfingerprints of a plurality of host signals are stored, an originalfingerprint (M(i)) which substantially matches the extractedfingerprint, and means (4) for determining the difference between theoriginal fingerprint found in the database and said extractedfingerprint, where said difference represents the watermark (w(i)) to beretrieved.